OneIO
OneIO is a Rust library that provides unified simple IO interface for reading and writing to and from data files from different sources and compressions.
Usage and Feature Flags
Enable all compression algorithms, and handle remote files (default)
= "0.10.1"
Select from supported feature flags
= { = "0.10.1", = ["remote", "gz"]}
Supported feature flags:
all
(default): all flags (["gz", "bz", "lz", "remote", "json", "s3"]
)remote
: allow reading from remote filesgz
: supportgzip
filesbz
: supportbzip2
fileslz
: supportlz4
filesjson
: allow reading JSON content into structs directlys3
: allow reading from AWS S3 compatible buckets
Use oneio
commandline tool
OneIO comes with a commandline tool, oneio
, that opens and reads local/remote files
to terminal and handles decompression automatically. This can be useful if you want
read some compressed plain-text files from a local or remote source.
Mingwei Zhang <mingwei@bgpkit.com>
OneIO is a Rust library that provides unified simple IO interface for
reading and writing to and from data files from different sources and compressions.
USAGE:
oneio [OPTIONS] <FILE>
ARGS:
<FILE> file to open, remote or local
OPTIONS:
--cache-dir <CACHE_DIR> cache reading to specified directory
--cache-file <CACHE_FILE> specify cache file name
--cache-force force re-caching if local cache already exists
-d, --download download the file to current directory, similar to run `wget`
-h, --help Print help information
-o, --outfile <OUTFILE> output file path
-s, --stats read through file and only print out stats
-V, --version Print version information
You can just specify a data file location after oneio
. The following command
prints out the raw HTML file from https://bgpkit.com.
Here is another example of using oneio
to read an remote compressed JSON file,
pipe it to jq
and count the number of JSON objects in the array.
|
You can also directly download a file with the --download
(or -d
) flag.
|
|||||||||||||
|||||||||||||
|||||||||||||
|||||||||||||
|||||||||||||
Use OneIO Reader as Library
The returned reader implements BufRead, and handles decompression from the following types:
gzip
: files ending withgz
orgzip
bzip2
: files ending withbz
orbz2
lz4
: files ending withlz4
orlz
It also handles reading from remote or local files transparently.
Examples
Read all into string:
const TEST_TEXT: &str = "OneIO test file.
This is a test.";
Read into lines:
use BufRead;
const TEST_TEXT: &str = "OneIO test file.
This is a test.";
Use OneIO Writer as Library
[get_writer] returns a generic writer that implements [Write], and handles decompression from the following types:
gzip
: files ending withgz
orgzip
bzip2
: files ending withbz
orbz2
Note: lz4 writer is not currently supported.
Example
Common IO operations
Read remote content with custom headers
use HashMap;
Download remote file to local directory
S3-related operations (needs s3
feature flag)